Precise Error Analysis of Regularized M-estimators in High-dimensions

A popular approach for estimating an unknown signal x0 ∈ R from noisy, linear measurements y = Ax0 +z ∈ R is via solving a so called regularized M-estimator: x̂ := arg minx L(y−Ax)+λf(x). Here, L is a convex loss function, f is a convex (typically, non-smooth) regularizer, and, λ > 0 is a regularizer parameter. We analyze the squared error performance ‖x̂ − x0‖2 of such estimators in the high-dimensional proportional regime where m,n → ∞ and m/n → δ. The design matrix A is assumed to have entries iid Gaussian; only minimal and rather mild regularity conditions are imposed on the loss function, the regularizer, and on the noise and signal distributions. We show that the squared error converges in probability to a nontrivial limit that is given as the solution to a minimax convex-concave optimization problem on four scalar optimization variables. We identify a new summary parameter, termed the Expected Moreau envelope to play a central role in the error characterization. The precise nature of the results permits an accurate performance comparison between different instances of regularized M-estimators and allows to optimally tune the involved parameters (e.g. regularizer parameter, number of measurements). The key ingredient of our proof is the Convex Gaussian Min-max Theorem (CGMT) which is a tight and strengthened version of a classical Gaussian comparison inequality that was proved by Gordon in 1988.